Introduction

What is this

This notebook will include informal meta-analyses of different metrics and methods for evaluating surgical skill.

The reported metrics compare differences between novices and expert surgeons.

It is informal because it’s not based on systematic review, and because some studies have been included with very relaxed conditions. For example, I have picked the novices and experts without comparing their definitions between studies. Novice = weakest skill group in the study, expert = strongest skill group in the study. If a study included more than 2 groups, I picked the weakest (=novice) and strongest (=expert) groups’ results and discarded the others. If a study included more than 1 task, or several sub-tasks, I picked the one with largest difference between groups.

Many papers did report means and standard deviations explicitly, so they had to be estimated from boxplots/barplots, or by some other means

For example, sometimes studies reported only mean or median, but no SE/SD. I estimated the SD/SE in those cases based e.g. on the SD of some other similar metric that they reported, or the SD of previous results for the same metric. See the excel file for notes on each study.

May or may not be turned into more systematic meta-analysis later.

Example metrics that will be most likely included (Bolded ones have priority)

  • Task time
  • Tool Path length
  • Tool Velocity
  • Tool Acceleration
  • Tool Curvature
  • Idle time
  • Pupil dilations
  • Blinks
  • Tool Movement efficiency
  • Number of movements
  • Tool Forces
  • Tool Torques
  • Bimanual dexterity
  • Jerk
  • Fixation duration
  • Saccade amplitudes
  • EEG?

Full list of papers and metrics can be found in the excel file shared in the repo:

Link to Github repo

Last update: 20.6.2022.

Submit results

If you notice errors or know some good studies to be included, feel free to forward them to

jani.koskinen [ at ] uef.fi

or use the form below TBD

How results are calculated

  1. From each study, extract
  • Number of trials per group (Nn, Ne, for novices and experts, respectively)
  • Means per group (Mn, Me for novices and experts, respectively)
  • Standard deviations per group (SDn, SDe)
  1. Calculate pooled standard deviation SDpooled
  2. Normalize by calculating Standardized Mean Difference (SDM): (Mn - Me)/SDpooled
  3. Calculate small sample size correction g = SMD*(1 - 3/(4n - 9)), where n is the total sample size of the study (both groups combined).
  4. Calculate SDg, standard deviation after correction

These values are used as input in the R meta package’s metagen function.

For more information, check:

Doing Meta-Analysis with R: A Hands-On Guide

Forest plots

Forest plot explanation

Sample size estimation

How many samples needed at some effect size d? At alpha = 0.05 and power = 0.8 and using t-test. Assuming independent trials (e.g. no multiple measurements from same participants etc.)

Hover mouse over the points in the plot to see the values. Some baseline effect sizes from the meta-analyses given as baseline:

IT = Idle Time

TT = Task Time

BD = Bimanual Dexterity

TEPR = Task-Evoked Pupil Reaction/Dilation (Esimated without one outlier study removed)

TJ = Tool Jerk

TF = Tool Forceg

Task Time

Task time is the time taken to complete a task. Task can be short like a single knot or some longer complex task.

Studies

Load data

df.time <- read_excel('data/surgical_metrics.xlsx', sheet='task_time')

Print studies

Author Year Study Journal
Koskinen et al. 2022 Utilizing Grasp Monitoring to Predict Microsurgical Expertise Journal of Surgical Research
Chainey et al. 2021 Eye-Hand Coordination of Neurosurgeons: Evidence of Action-Related Fixation in Microsuturing World Neurosurgery
Harada et al. 2015 Assessing microneurosurgical skill with medico-engineering technology World Neurosurgery
Vedula et al. 2016 Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment Journal of Surgical Education
Judkins et al. 2009 Objective evaluation of expert and novice performance during robotic surgical training tasks Surgical Endoscopy
Smith et al. 2002 Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy Surgical Endoscopy and Other Interventional Techniques
Francis et al. 2002 The performance of master surgeons on the Advanced Dundee Endoscopic Psychomotor Tester: Contrast validity study Archives of Surgery
Moorthy et al. 2004 Bimodal assessment of laparoscopic suturing skills: Construct and concurrent validity Surgical Endoscopy and Other Interventional Techniques
Van Sickle et al. 2008 Construct validity of an objective assessment method for laparoscopic intracorporeal suturing and knot tying The American Journal of Surgery
Xeroulis et al. 2009 Simulation in laparoscopic surgery: A concurrent validity study for FLS Surgical Endoscopy and Other Interventional Techniques
Huffman et al. 2020 Optimizing Assessment of Surgical Knot Tying Skill Journal of Surgical Education
Law et al. 2004 Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment Proceedings of the Eye tracking research & applications symposium on Eye tracking research & applications - ETRA’2004
Kazemi et al. 2010 Assessing suturing techniques using a virtual reality surgical simulator Microsurgery
O’Toole et al. 1999 Measuring and Developing Suturing Technique with a Virtual Reality Surgical Simulator Journal ofthe American College of Surgeons
Zheng et al. 2021 Action-related eye measures to assess surgical expertise BJS Open
Datta et al. 2001 The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model Journal of the American College of Surgeons
Pagador et al. 2012 Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment International Journal of Computer Assisted Radiology and Surgery
Aggarwal et al. 2007 An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room Annals of Surgery
Wilson et al. 2010 Psychomotor control in a virtual laparoscopic surgery training environment: Gaze control parameters differentiate novices from experts Surgical Endoscopy
Hofstad et al. 2013 A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance Surgical Endoscopy and Other Interventional Techniques
Hung et al. 2018 Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study Journal of Urology
Yamaguchi et al. 2011 Objective assessment of laparoscopic suturing skills using a motion-tracking system Surgical Endoscopy
Pellen et al. 2009 Laparoscopic surgical skills assessment: Can simulators replace experts? World Journal of Surgery
Pastewski et al. 2021 Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills Journal of Surgical Education
Chmarra et al. 2010 Objective classification of residents based on their psychomotor laparoscopic skills Surgical Endoscopy and Other Interventional Techniques

Results

m.time <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.time,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Time to completion in Surgery")
summary(m.time)
## Review:     Time to completion in Surgery
## 
##                      SMD             95%-CI %W(random)
## Koskinen et al.   1.8413 [ 1.4135;  2.2691]        4.5
## Chainey et al.    0.7034 [ 0.0383;  1.3686]        4.4
## Harada et al.     1.5503 [ 0.8551;  2.2456]        4.4
## Vedula et al.     2.2149 [ 1.7299;  2.6999]        4.5
## Judkins et al.    5.3971 [ 3.8216;  6.9726]        3.7
## Smith et al.      8.0559 [ 5.5551; 10.5568]        2.9
## Francis et al.    0.9801 [ 0.3227;  1.6375]        4.4
## Moorthy et al.    1.4157 [ 0.3397;  2.4917]        4.1
## Van Sickle et al. 2.1365 [ 1.0202;  3.2528]        4.1
## Xeroulis et al.   2.5525 [ 1.2650;  3.8400]        4.0
## Huffman et al.    6.5116 [ 4.9174;  8.1059]        3.7
## Law et al.        2.0257 [ 1.3401;  2.7112]        4.4
## Kazemi et al.     0.8354 [-0.3084;  1.9791]        4.1
## O'Toole et al.    1.7086 [ 0.6569;  2.7602]        4.1
## Zheng et al.      1.9382 [ 0.6894;  3.1871]        4.0
## Datta et al.      2.1791 [ 1.1762;  3.1819]        4.2
## Pagador et al.    6.3695 [ 2.5221; 10.2170]        2.0
## Aggarwal et al.   0.1873 [-0.4390;  0.8136]        4.4
## Wilson et al.     1.3349 [ 0.1520;  2.5179]        4.1
## Hofstad et al.    1.3791 [ 0.3199;  2.4382]        4.1
## Hung et al.       2.2342 [ 1.7203;  2.7481]        4.5
## Yamaguchi et al.  4.5870 [ 2.7625;  6.4116]        3.5
## Pellen et al.     5.6362 [ 3.6128;  7.6596]        3.3
## Pastewski et al.  0.5600 [-0.1171;  1.2371]        4.4
## Chmarra et al.    0.9810 [ 0.0706;  1.8914]        4.2
## 
## Number of studies combined: k = 25
## 
##                         SMD           95%-CI    t  p-value
## Random effects model 2.3748 [1.5627; 3.1869] 6.04 < 0.0001
## 
## Quantifying heterogeneity:
##  tau^2 = 2.9577 [1.8111; 7.7816]; tau = 1.7198 [1.3458; 2.7895]
##  I^2 = 86.4% [81.1%; 90.2%]; H = 2.71 [2.30; 3.19]
## 
## Test of heterogeneity:
##       Q d.f.  p-value
##  176.04   24 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.time,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Time to completion in Surgery")

#dev.print(pdf, "figures/forest_time.pdf", width=10, height=10)

Discussion

Time to completion is by far the most often reported metric. It is often reported even when it is not the main focus of the study.

Bimanual Dexterity

Bimanual dexterity is a measure of how well the surgeon is able to use both hands at the same time. Note that there are many different ways for calculating “ability to use both hands simultaneously.”

Studies

Load data

df.biman <- read_excel('data/surgical_metrics.xlsx', sheet='tool_bimanual')

Print studies

Author Year Study Journal
Koskinen et al. 2022 Movement-level process modeling of microsurgical bimanual and unimanual tasks International Journal of Computer Assisted Radiology and Surgery
Hofstad et al. 2017 Psychomotor skills assessment by motion analysis in minimally invasive surgery on an animal organ Minimally Invasive Therapy and Allied Technologies
Demirel et al. 2022 Scoring metrics for assessing skills in arthroscopic rotator cuff repair: performance comparison study of novice and expert surgeons International Journal of Computer Assisted Radiology and Surgery
Islam et al. 2016 Affordable, web-based surgical skill training and evaluation tool Journal of Biomedical Informatics
Zulbaran-Rojas et al. 2021 Utilization of Flexible-Wearable Sensors to Describe the Kinematics of Surgical Proficiency Journal of Surgical Research
Mori et al. 2022 Validation of a novel virtual reality simulation system with the focus on training for surgical dissection during laparoscopic sigmoid colectomy BMC Surgery

Results

Run meta-analysis

m.biman <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.biman,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Bimanual dexterity in Surgery")

Print results

summary(m.biman)
## Review:     Bimanual dexterity in Surgery
## 
##                           SMD              95%-CI %W(random)
## Koskinen et al.       -3.0589 [ -3.8825; -2.2353]       17.4
## Hofstad et al.        -3.0127 [ -4.6473; -1.3782]       16.0
## Demirel et al.        -2.0314 [ -3.0378; -1.0251]       17.1
## Islam et al.          -8.6969 [-10.7900; -6.6039]       15.1
## Zulbaran-Rojas et al. -0.8250 [ -1.7586;  0.1085]       17.2
## Mori et al.           -2.6867 [ -3.6936; -1.6799]       17.1
## 
## Number of studies combined: k = 6
## 
##                          SMD             95%-CI     t p-value
## Random effects model -3.2752 [-6.0448; -0.5057] -3.04  0.0288
## 
## Quantifying heterogeneity:
##  tau^2 = 6.0648 [2.0109; 43.9963]; tau = 2.4627 [1.4181; 6.6330]
##  I^2 = 89.8% [80.5%; 94.7%]; H = 3.13 [2.27; 4.33]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  49.11    5 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.biman,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Bimanual dexterity in Surgery")

#dev.print(pdf, "figures/forest_biman.pdf", width=8, height=8)

Discussion

Analysis of bimanual dexterity is made harder because there are so many different definitions for it.

Tool Movements

Number of tool movements made during the task. Note: I have included here the grasp results from our paper (and other studies that analyzed only one type of action/movement)

Studies

Load data

df.toolmvt <- read_excel('data/surgical_metrics.xlsx', sheet='tool_movements')

Print studies

Author Year Study Journal
Datta et al. 2001 The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model Journal of the American College of Surgeons
Pagador et al. 2012 Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment International Journal of Computer Assisted Radiology and Surgery
Koskinen et al. 2022 Utilizing Grasp Monitoring to Predict Microsurgical Expertise Journal of Surgical Research
Bann et al. 2003 Measurement of surgical dexterity using motion analysis of simple bench tasks World Journal of Surgery
Smith et al. 2002 Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy Surgical Endoscopy and Other Interventional Techniques
Aggarwal et al. 2007 An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room Annals of Surgery
Yamaguchi et al. 2007 Construct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulator Surgical Endoscopy and Other Interventional Techniques
Goldbraikh et al. 2021 Video-based fully automatic assessment of open surgery suturing skills International Journal of Computer Assisted Radiology and Surgery
Vedula et al. 2016 Task-Level vs . Segment-Level Quantitative Metrics for Surgical Skill Assessment Journal of Surgical Education
Wilson et al. 2010 Psychomotor control in a virtual laparoscopic surgery training environment: Gaze control parameters differentiate novices from experts Surgical Endoscopy
Hofstad et al. 2013 A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance Surgical Endoscopy and Other Interventional Techniques

Results

Run meta-analysis

m.toolmvt <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolmvt,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool movements in Surgery")
summary(m.toolmvt)
## Review:     Tool movements in Surgery
## 
##                       SMD             95%-CI %W(random)
## Datta et al.       2.0390 [ 1.0607;  3.0174]        9.6
## Pagador et al.    10.0866 [ 4.2364; 15.9368]        3.7
## Koskinen et al.    1.3393 [ 0.7781;  1.9006]       10.0
## Bann et al.        1.2504 [ 0.4629;  2.0380]        9.8
## Smith et al.       5.9403 [ 4.0136;  7.8671]        8.5
## Aggarwal et al.   -0.0641 [-0.6894;  0.5612]        9.9
## Yamaguchi et al.   2.3074 [ 1.3887;  3.2260]        9.7
## Goldbraikh et al.  2.6143 [ 1.8535;  3.3751]        9.8
## Vedula et al.      6.5233 [ 5.6418;  7.4047]        9.7
## Wilson et al.      0.5955 [-0.4889;  1.6799]        9.5
## Hofstad et al.     0.9586 [-0.0444;  1.9617]        9.6
## 
## Number of studies combined: k = 11
## 
##                         SMD           95%-CI    t p-value
## Random effects model 2.5911 [0.8299; 4.3523] 3.28  0.0083
## 
## Quantifying heterogeneity:
##  tau^2 = 5.1496 [2.4728; 26.0028]; tau = 2.2693 [1.5725; 5.0993]
##  I^2 = 94.7% [92.2%; 96.4%]; H = 4.33 [3.58; 5.24]
## 
## Test of heterogeneity:
##       Q d.f.  p-value
##  187.46   10 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolmvt,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool movements in Surgery")

#dev.print(pdf, "figures/forest_toolmvt.pdf", width=8, height=8)

Discussion

Tool movements are perhaps the second most often reported metric. Different papers measure, analyze and report them differently. Often connected to “movement efficiency”.

Tool Idle Time

Tool idle time measures how long the tools were not being used, either as time or as fraction of the complete task time.

Studies

Load data

df.toolidle <- read_excel('data/surgical_metrics.xlsx', sheet='tool_idle')

Print studies

Author Year Study Journal
Koskinen et al. 2021 Movement-level process modeling of microsurgical bimanual and unimanual tasks International Journal of Computer Assisted Radiology and Surgery
Uemura et al. 2015 Procedural surgical skill assessment in laparoscopic training environments International Journal of Computer Assisted Radiology and Surgery
D’Angelo et al. 2015 Idle time: An underdeveloped performance metric for assessing surgical skill American Journal of Surgery
Mackenzie et al. 2021 Enhanced Training Benefits of Video Recording Surgery With Automated Hand Motion Analysis World Journal of Surgery
Oropesa et al. 2013 Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery Surgical Innovation
Hung et al. 2018 Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study Journal of Urology
Topalli et al. 2018 Eye-Hand Coordination Patterns of Intermediate and Novice Surgeons in a Simulation-Based Endoscopic Surgery Training Environment Journal of Eye Movement Research

Results

Run meta-analysis

m.toolidle <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolidle,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Idle time in Surgery")
summary(m.toolidle)
## Review:     Idle time in Surgery
## 
##                     SMD            95%-CI %W(random)
## Koskinen et al.  2.5600 [ 1.8069; 3.3131]       17.7
## Uemura et al.    1.6933 [ 0.7816; 2.6050]       16.2
## D'Angelo et al.  2.7724 [ 1.0363; 4.5085]        9.3
## Mackenzie et al. 0.3642 [-1.6449; 2.3734]        7.7
## Oropesa et al.   0.9556 [-0.1828; 2.0941]       14.0
## Hung et al.      0.6990 [ 0.2837; 1.1144]       20.8
## Topalli et al.   0.6159 [-0.4828; 1.7147]       14.3
## 
## Number of studies combined: k = 7
## 
##                         SMD           95%-CI    t p-value
## Random effects model 1.3803 [0.5279; 2.2326] 3.96  0.0074
## 
## Quantifying heterogeneity:
##  tau^2 = 0.5520 [0.0915; 4.0820]; tau = 0.7430 [0.3024; 2.0204]
##  I^2 = 75.3% [47.7%; 88.3%]; H = 2.01 [1.38; 2.93]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  24.29    6  0.0005
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolidle,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Idle time in Surgery")

#dev.print(pdf, "figures/forest_toolidle.pdf", width=8, height=8)

Discussion

Not many papers that focused on idle time.

Tool Path Length

How much the tools travel during the task.

Studies

Load data

df.toolpl <- read_excel('data/surgical_metrics.xlsx', sheet='tool_path_length')

Print studies

Author Year Study Journal
Aggarwal et al. 2007 An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room Annals of Surgery
Moorthy et al. 2004 Bimodal assessment of laparoscopic suturing skills: Construct and concurrent validity Surgical Endoscopy and Other Interventional Techniques
Smith et al. 2002 Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy Surgical Endoscopy and Other Interventional Techniques
Pagador et al. 2012 Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment International Journal of Computer Assisted Radiology and Surgery
Goldbraikh et al. 2021 Video-based fully automatic assessment of open surgery suturing skills International Journal of Computer Assisted Radiology and Surgery
Jimbo et al. 2017 A new innovative laparoscopic fundoplication training simulator with a surgical skill validation system Surgical Endoscopy
Hofstad et al. 2013 A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance Surgical Endoscopy and Other Interventional Techniques
Oropesa et al. 2013 Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery Surgical Innovation
Pellen et al. 2009 Laparoscopic surgical skills assessment: Can simulators replace experts? World Journal of Surgery
D’Angelo et al. 2015 Idle time: An underdeveloped performance metric for assessing surgical skill American Journal of Surgery
Hung et al. 2018 Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study Journal of Urology
Vedula et al. 2016 Task-Level vs . Segment-Level Quantitative Metrics for Surgical Skill Assessment Journal of Surgical Education
Yamaguchi et al. 2011 Objective assessment of laparoscopic suturing skills using a motion-tracking system Surgical Endoscopy
Harada et al. 2015 Assessing Microneurosurgical Skill with Medico-Engineering Technology World Neurosurgery
Ebina et al. 2021 Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs Surgical Endoscopy
Chmarra et al. 2010 Objective classification of residents based on their psychomotor laparoscopic skills Surgical Endoscopy and Other Interventional Techniques

Results

Run meta-analysis

m.toolpl <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolpl,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool path length in Surgery")
summary(m.toolpl)
## Review:     Tool path length in Surgery
## 
##                      SMD             95%-CI %W(random)
## Aggarwal et al.   0.0647 [-0.5606;  0.6900]        7.6
## Moorthy et al.    1.3161 [ 0.2542;  2.3780]        6.0
## Smith et al.      2.6541 [ 1.5194;  3.7889]        5.7
## Pagador et al.    6.2865 [ 2.4827; 10.0904]        1.2
## Goldbraikh et al. 2.0174 [ 1.3325;  2.7024]        7.4
## Jimbo et al.      0.8695 [ 0.1950;  1.5441]        7.5
## Hofstad et al.    0.7989 [-0.1876;  1.7853]        6.3
## Oropesa et al.    0.2889 [-0.8108;  1.3885]        5.8
## Pellen et al.     2.0960 [ 0.9877;  3.2042]        5.8
## D'Angelo et al.   1.7506 [ 0.3193;  3.1819]        4.7
## Hung et al.       1.8000 [ 1.3220;  2.2779]        8.1
## Vedula et al.     2.4204 [ 1.9215;  2.9194]        8.1
## Yamaguchi et al.  3.3661 [ 1.8874;  4.8449]        4.5
## Harada et al.     1.0214 [ 0.3743;  1.6685]        7.6
## Ebina et al.      0.9071 [ 0.1861;  1.6281]        7.3
## Chmarra et al.    1.2076 [ 0.2706;  2.1447]        6.4
## 
## Number of studies combined: k = 16
## 
##                         SMD           95%-CI    t  p-value
## Random effects model 1.5109 [0.9657; 2.0561] 5.91 < 0.0001
## 
## Quantifying heterogeneity:
##  tau^2 = 0.5801 [0.2782; 3.3711]; tau = 0.7617 [0.5275; 1.8360]
##  I^2 = 78.4% [65.5%; 86.5%]; H = 2.15 [1.70; 2.72]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  69.48   15 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolpl,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool path length in Surgery")

#dev.print(pdf, "figures/forest_toolpl.pdf", width=8, height=8)

Discussion

Tool path length also a very common metric. Most studies report that novices have much larger path length, indicating less effective movements. Results differ based on task and surgical

Tool Velocity

Tool velocity/speed measures how fast the surgical tool or tools are moving.

Studies

Load data

df.toolvelocity <- read_excel('data/surgical_metrics.xlsx', sheet='tool_velocity')

Print studies

Author Year Study Journal
Davids et al. 2021 Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation. World Neurosurgery
Pastewski et al. 2021 Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills Journal of Surgical Education
Hwang et al. 2006 Correlating motor performance with surgical error in laparoscopic cholecystectomy Surgical Endoscopy and Other Interventional Techniques
Ebina et al. 2021 Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs Surgical Endoscopy
Jimbo et al. 2017 A new innovative laparoscopic fundoplication training simulator with a surgical skill validation system Surgical Endoscopy
Judkins et al. 2009 Objective evaluation of expert and novice performance during robotic surgical training tasks Surgical Endoscopy
Hofstad et al. 2013 A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance Surgical Endoscopy and Other Interventional Techniques
Frasier et al. 2016 A marker-less technique for measuring kinematics in the operating room Surgery (United States)
Azari et al. 2018 Can surgical performance for varying experience be measured from hand motions? Proceedings of the Human Factors and Ergonomics Society
Pagador et al. 2012 Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment International Journal of Computer Assisted Radiology and Surgery

Results

Run meta-analysis

m.toolvelocity <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolvelocity,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool velocity in Surgery")
summary(m.toolvelocity)
## Review:     Tool velocity in Surgery
## 
##                      SMD             95%-CI %W(random)
## Davids et al.     0.5140 [-1.5370;  2.5650]        5.0
## Pastewski et al. -0.7177 [-1.4028; -0.0326]       12.8
## Hwang et al.      6.1176 [ 1.5045; 10.7307]        1.3
## Ebina et al.     -0.8684 [-1.5865; -0.1503]       12.5
## Jimbo et al.     -0.7654 [-1.4334; -0.0974]       12.9
## Judkins et al.    0.6675 [-0.0690;  1.4039]       12.4
## Hofstad et al.    1.0086 [-0.0002;  2.0174]       10.4
## Frasier et al.   -1.1447 [-1.7143; -0.5751]       13.6
## Azari et al.     -0.2982 [-1.2042;  0.6078]       11.1
## Pagador et al.    0.0585 [-1.3278;  1.4448]        7.9
## 
## Number of studies combined: k = 10
## 
##                          SMD            95%-CI     t p-value
## Random effects model -0.1917 [-0.9642; 0.5809] -0.56  0.5883
## 
## Quantifying heterogeneity:
##  tau^2 = 0.4917 [0.1990; 10.7136]; tau = 0.7012 [0.4461; 3.2732]
##  I^2 = 74.0% [51.2%; 86.1%]; H = 1.96 [1.43; 2.68]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  34.58    9 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolvelocity,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool velocity in Surgery")

#dev.print(pdf, "figures/forest_toolvelocity.pdf", width=8, height=8)

Discussion

Velocity (and related metrics like acceleration) are semi-popular method. Results seem to vary a lot, sometimes novices are faster and sometimes experts are faster. May depend on task?

Tool Acceleration

Tool acceleration measures how much the tool/tools accelerate during the task.

Studies

Load data

df.toolacc <- read_excel('data/surgical_metrics.xlsx', sheet='tool_acceleration')

Print studies

Author Year Study Journal
Azari et al. 2018 Can surgical performance for varying experience be measured from hand motions? Proceedings of the Human Factors and Ergonomics Society
Frasier et al. 2016 A marker-less technique for measuring kinematics in the operating room Surgery (United States)
Ebina et al. 2021 Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs Surgical Endoscopy
Pastewski et al. 2021 Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills Journal of Surgical Education
Davids et al. 2021 Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation. World Neurosurgery

Results

Run meta-analysis

m.toolacc <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolacc,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool acceleration in Surgery")
summary(m.toolacc)
## Review:     Tool acceleration in Surgery
## 
##                      SMD             95%-CI %W(random)
## Azari et al.     -0.3713 [-1.2803;  0.5377]       18.5
## Frasier et al.   -1.0298 [-1.5922; -0.4674]       27.7
## Ebina et al.     -0.7891 [-1.5016; -0.0767]       23.3
## Pastewski et al.  0.1911 [-0.4748;  0.8570]       24.7
## Davids et al.    -0.0233 [-2.0633;  2.0167]        5.8
## 
## Number of studies combined: k = 5
## 
##                          SMD            95%-CI     t p-value
## Random effects model -0.4926 [-1.1602; 0.1750] -2.05  0.1099
## 
## Quantifying heterogeneity:
##  tau^2 = 0.1828 [0.0000; 1.9388]; tau = 0.4275 [0.0000; 1.3924]
##  I^2 = 52.3% [0.0%; 82.5%]; H = 1.45 [1.00; 2.39]
## 
## Test of heterogeneity:
##     Q d.f. p-value
##  8.39    4  0.0782
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolacc,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool acceleration in Surgery")

#dev.print(pdf, "figures/forest_toolacceleration.pdf", width=8, height=8)

Discussion

Not many papers that focused on tool accelerations. Jerk (third derivative of position, derivative of acceleration) is much more popular.

Tool Jerk

Jerk is the third derivative of the surgical instruments position, and measures how smooth the movements are.

Studies

Load data

df.jerk <- read_excel('data/surgical_metrics.xlsx', sheet='tool_jerk')

Print studies

Author Year Study Journal
Ghasemloonia et al. 2017 Surgical Skill Assessment Using Motion Quality and Smoothness Journal of Surgical Education
Hwang et al. 2006 Correlating motor performance with surgical error in laparoscopic cholecystectomy Surgical Endoscopy and Other Interventional Techniques
Ebina et al. 2021 Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs Surgical Endoscopy
Azari et al. 2018 Can surgical performance for varying experience be measured from hand motions? Proceedings of the Human Factors and Ergonomics Society
Davids et al. 2021 Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation. World Neurosurgery
Oropesa et al. 2013 Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery Surgical Innovation
Maithel et al 2005 Simulated laparoscopy using a head-mounted display vs traditional video monitor: An assessment of performance and muscle fatigue Surgical Endoscopy and Other Interventional Techniques
Liang et al. 2018 Motion control skill assessment based on kinematic analysis of robotic end-effector movements The International Journal of Medical Robotics and Computer Assisted Surgery
Islam et al. 2016 Affordable, web-based surgical skill training and evaluation tool Journal of Biomedical Informatics
Hofstad et al. 2017 Psychomotor skills assessment by motion analysis in minimally invasive surgery on an animal organ Minimally Invasive Therapy and Allied Technologies
Shafiel et al. 2017 Motor Skill Evaluation During Robot-Assisted Surgery Volume 5A: 41st Mechanisms and Robotics Conference
Chmarra et al. 2010 Objective classification of residents based on their psychomotor laparoscopic skills Surgical Endoscopy and Other Interventional Techniques

Results

Run meta-analysis

m.jerk <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.jerk,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Jerk in Surgery")
summary(m.jerk)
## Review:     Jerk in Surgery
## 
##                         SMD             95%-CI %W(random)
## Ghasemloonia et al.  1.7090 [ 1.1677;  2.2504]        9.6
## Hwang et al.         2.6183 [ 0.1709;  5.0658]        5.0
## Ebina et al.        -0.9365 [-1.6598; -0.2133]        9.3
## Azari et al.        -0.1972 [-0.8043;  0.4098]        9.5
## Davids et al.        0.1307 [-1.9101;  2.1714]        5.9
## Oropesa et al.      -0.9775 [-2.1179;  0.1629]        8.3
## Maithel et al        1.6060 [ 0.7461;  2.4658]        9.0
## Liang et al.         0.1596 [-0.7184;  1.0377]        8.9
## Islam et al.         3.6094 [ 2.4907;  4.7280]        8.3
## Hofstad et al.       1.3201 [-0.1549;  2.7952]        7.4
## Shafiel et al.       0.4174 [ 0.2951;  0.5397]       10.1
## Chmarra et al.       0.8995 [-0.0026;  1.8015]        8.9
## 
## Number of studies combined: k = 12
## 
##                         SMD            95%-CI    t p-value
## Random effects model 0.7932 [-0.0699; 1.6564] 2.02  0.0681
## 
## Quantifying heterogeneity:
##  tau^2 = 1.5140 [0.6120; 5.0878]; tau = 1.2305 [0.7823; 2.2556]
##  I^2 = 87.6% [80.2%; 92.2%]; H = 2.84 [2.25; 3.59]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  88.85   11 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.jerk,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Jerk in Surgery")

#dev.print(pdf, "figures/forest_tooljerk.pdf", width=8, height=8)

Discussion

TBD

Tool Force

Tool force is the force the surgeon uses when they e.g. grasp something using the surgical tools.

Studies

Load data

df.force <- read_excel('data/surgical_metrics.xlsx', sheet='tool_force')

Print studies

Author Year Study Journal
Harada et al. 2015 Assessing Microneurosurgical Skill with Medico-Engineering Technology World Neurosurgery
Prasad et al. 2016 Objective Assessment of Laparoscopic Force and Psychomotor Skills in a Novel Virtual Reality-Based Haptic Simulator Journal of Surgical Education
Horeman et al. 2014 Assessment of Laparoscopic Skills Based on Force and Motion Parameters IEEE Transactions on Biomedical Engineering
Trejos et al. 2014 Development of force-based metrics for skills assessment in minimally invasive surgery Surgical Endoscopy
Woodrow et al. 2007 Training and evaluating spinal surgeons: The development of novel performance measures Spine
Sugiyama et al. 2018 Forces of Tool-Tissue Interaction to Assess Surgical Skill Level JAMA Surgery

Results

Run meta-analysis

m.force <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.force,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Force use in Surgery")
summary(m.force)
## Review:     Force use in Surgery
## 
##                    SMD            95%-CI %W(random)
## Harada et al.   0.5357 [-0.0830; 1.1545]       19.3
## Prasad et al.   1.2450 [ 0.6378; 1.8523]       19.3
## Horeman et al.  2.7082 [ 1.5555; 3.8609]       16.3
## Trejos et al.   1.5351 [ 0.2224; 2.8477]       15.3
## Woodrow et al.  3.8205 [ 2.2438; 5.3972]       13.7
## Sugiyama et al. 0.3071 [-0.8880; 1.5022]       16.0
## 
## Number of studies combined: k = 6
## 
##                         SMD           95%-CI    t p-value
## Random effects model 1.5936 [0.2394; 2.9477] 3.03  0.0292
## 
## Quantifying heterogeneity:
##  tau^2 = 1.2415 [0.2843; 10.4203]; tau = 1.1142 [0.5332; 3.2280]
##  I^2 = 79.2% [54.5%; 90.5%]; H = 2.19 [1.48; 3.24]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  24.02    5  0.0002
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.force,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Force use in Surgery")

#dev.print(pdf, "figures/forest_toolforce.pdf", width=8, height=8)

Discussion

Forces analyzed somewhat commonly, but often not between novices and experts, but within tasks, or tools, or skill groups.

Task-Evoked Pupil Dilation

Pupil size measures cognitive workload, stress, and million other things.

Studies

Load data

df.pupil <- read_excel('data/surgical_metrics.xlsx', sheet='pupil_dilation')

Print studies

Author Year Study Journal
Castner et al. 2020 Pupil diameter differentiates expertise in dental radiography visual search PLOS ONE
Cabrera-Mino et al. 2019 Task-Evoked Pupillary Responses in Nursing Simulation as an Indicator of Stress and Cognitive Load Clinical Simulation in Nursing
Bednarik et al. 2018 Pupil Size As an Indicator of Visual-motor Workload and Expertise in Microsurgical Training Tasks Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications
Gunawardena et al. 2019 Assessing Surgeons’ Skill Level in Laparoscopic Cholecystectomy using Eye Metrics Eye Tracking Research and Applications Symposium (ETRA)
Dilley et al. 2020 Visual behaviour in robotic surgery—Demonstrating the validity of the simulated environment International Journal of Medical Robotics and Computer Assisted Surgery
Gao et al. 2018 Quantitative evaluations of the effects of noise on mental workloads based on pupil dilation during laparoscopic surgery American Surgeon

Results

Run meta-analysis

m.pupil <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.pupil,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Pupil dilation in Surgery")
summary(m.pupil)
## Review:     Pupil dilation in Surgery
## 
##                         SMD             95%-CI %W(random)
## Castner et al.       0.7877 [ 0.6671;  0.9083]       17.6
## Cabrera-Mino et al.  0.8255 [ 0.0502;  1.6009]       16.7
## Bednarik et al.     -2.9791 [-3.5250; -2.4332]       17.1
## Gunawardena et al.   1.5927 [ 0.3701;  2.8152]       15.4
## Dilley et al.       -0.0152 [-0.7136;  0.6833]       16.8
## Gao et al.           1.2184 [ 0.3422;  2.0946]       16.4
## 
## Number of studies combined: k = 6
## 
##                         SMD            95%-CI    t p-value
## Random effects model 0.2082 [-1.5479; 1.9643] 0.30  0.7728
## 
## Quantifying heterogeneity:
##  tau^2 = 2.6787 [0.9588; 16.5443]; tau = 1.6367 [0.9792; 4.0675]
##  I^2 = 97.3% [95.7%; 98.2%]; H = 6.03 [4.85; 7.51]
## 
## Test of heterogeneity:
##       Q d.f.  p-value
##  182.07    5 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.pupil,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Pupil dilation in Surgery")

#dev.print(pdf, "figures/forest_pupil.pdf", width=8, height=8)

Discussion

Prior research indicates that higher stress/cognitive workload -> larger pupil size. This is seen in most studies. In Bednarik et al. (2018), the effect is reversed. For that study, I picked needle piercing segment (because it was quaranteed to have un-interrupted visual contact from the participant). It can be that experts focused more on this, and had larger cognitive workload and pupil dilations.

Not that many studies that have measured pupil dilations and compared surgical novices and experts directly. Some used measures like ICA or Entropy (not included here). Pupil dilations used in other fields more often.

Tool Grasps

Grasps are the number of times the surgeon had to grasp something using surgical instruments

Studies

Load data

df.grasp <- read_excel('data/surgical_metrics.xlsx', sheet='tool_grasps')

Print studies

Author Year Study Journal
Koskinen et al. 2022 Utilizing Grasp Monitoring to Predict Microsurgical Expertise Journal of Surgical Research
Cao et al. 1996 Task and Motion Analysis in Endoscopic Surgery 5th Annual Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems

Results

Run meta-analysis

m.grasp <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.grasp,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Grasping in Surgery")
summary(m.grasp)
## Review:     Grasping in Surgery
## 
##                    SMD            95%-CI %W(random)
## Koskinen et al. 1.3393 [ 0.7781; 1.9006]       94.2
## Cao et al.      1.0000 [-1.2605; 3.2605]        5.8
## 
## Number of studies combined: k = 2
## 
##                         SMD           95%-CI     t p-value
## Random effects model 1.3196 [0.3112; 2.3280] 16.63  0.0382
## 
## Quantifying heterogeneity:
##  tau^2 = 0; tau = 0; I^2 = 0.0%; H = 1.00
## 
## Test of heterogeneity:
##     Q d.f. p-value
##  0.08    1  0.7752
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.grasp,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Grasping in Surgery")

#dev.print(pdf, "figures/forest_grasp.pdf", width=8, height=8)

Discussion

Not many papers that focused on grasps. I have included thus the grasp results also to the “Tool movements” analysis.